Tracking People and Recognizing Their Activities

We present a system for automatic people tracking and activity recognition. This video includes the contribution of 3 papers over 2 years [1], [2], [3]. We show results on many sequences, including clips from a feature-length film and historical sports footage (all tracked completely automatically). We show results that utilize the tracker to obtain 3D reconstructions and activity descriptions (also done automatically). Our basic approach to people-tracking is to build an appearance model for the person(s) in the video [1]. Since we do not know the appearance of a person a priori, we build the models on-the-fly. The video illustrates our method of using a stylized-pose detector (one that only finds people in lateral-walking configurations) [2]. Since people do not always walk laterally, we expect the detector to only fire in a few frames in a video sequence (shown in red). Our system builds a model of limb appearance from those sparse stylized detections (e.g., it might learn a baseball player has red arms and white legs). Our algorithm then re-processes the video, using the learned appearance models to find people in unrestricted configurations (shown in color). We can use our tracker to recover 3D configurations and activity labels. We assume we have a motion capture library where the 3D poses have been labeled off-line with activity descriptions. We then synthesize 3D motions that look like our tracks by matching short clips of 3D motion to short clips of the 2D tracks [3]. The synthesized motions are constructed from the original pool of labeled motions and thus are labeled themselves.